docs(self-hosted): provide more insights on troubleshooting kafka #15131
Conversation
Turns out most of our self-hosted users has never touch Kafka before, so it's a good idea to introduce them regarding how Kafka works. Also added how to increase consumers replica if they're lagging behind.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
![seer-by-sentry[bot]](https://avatars.githubusercontent.com/in/801464?s=60&v=4){kind=small}
Co-authored-by: Kevin Pfeifer <[email protected]>
Co-authored-by: Kevin Pfeifer <[email protected]>
| This section is aimed for those who have Kafka problems, but are not yet familiar with Kafka. At a high level, it is a message broker which stores message in a log (or in an easier language: very similar to an array) format. It receives messages from producers that aimed to a specific topic, and then sends them to consumers that are subscribed to that topic. The consumers can then process the messages. | ||
|
|
||
| This happens where Kafka and the consumers get out of sync. Possible reasons are: | ||
| On the inside, when a message enters a topic, it would be written to a certain partition. You can think partition as physical boxes that stores messages for a specific topic, each topic will have their own separate & dedicated partitions. In a distributed Kafka setup, each partition might be stored on a different machine/node, but if you only have a single Kafka instance, then all the partitions are stored on the same machine. |
There was a problem hiding this comment.
This paragraph has a bit of overlap with the next one and does not add much to understanding kafka IMO, so I think we could remove it completely.
There was a problem hiding this comment.
I don't mind leaving this here. At least I want to emphasize the existence of "partition"
Co-authored-by: Joris Bayer <[email protected]>
Co-authored-by: Joris Bayer <[email protected]>
| ```log | ||
| Exception: KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"} | ||
| ``` | ||
| This section is aimed for those who have Kafka problems, but are not yet familiar with Kafka. At a high level, Kafka is a message broker which stores messages in a log (or in an easier language: very similar to an array) format. It receives messages from producers that write to a specific topic, and then sends them to consumers that are subscribed to that topic. The consumers can then process the messages. |
There was a problem hiding this comment.
If you want to be pedantic, Kafka doesn't send messages to consumers. Consumers poll kafka and fetch new messages.
There was a problem hiding this comment.
Too pedantic and not so beginner friendly 😅
| 1. Running out of disk space or memory | ||
| 2. Having a sustained event spike that causes very long processing times, causing Kafka to drop messages as they go past the retention time | ||
| 3. Date/time out of sync issues due to a restart or suspend/resume cycle | ||
| When a producer sends a message to a topic, it will either stick to a certain partition number (example: partition 1, partition 2, etc.) or it will randomly choose a partition. A consumer will then subscribe to a topic and will automatically be assigned to one or more partitions by Kafka. The consumer will then start receiving messages from the assigned partitions. One very important aspect to note is that **the number of consumers within a consumer group must not exceed the number of partition for a given topic**. If you have more consumers than number of partitions, then the consumers will be hanging with no messages to consume. |
There was a problem hiding this comment.
If we use semantic partitioning, we don't generally assign partition numbers to messages. Instead we use 'keyed messages', which define how messages are grouped (by key) but how those keys map to partitions is up to Kafka.
There was a problem hiding this comment.
Adding on this, if messages don't have partition keys, kafka will assign the message to a partition via round-robin (so technically not randomly)
There was a problem hiding this comment.
Good point. Will update these.
| 3. Date/time out of sync issues due to a restart or suspend/resume cycle | ||
| When a producer sends a message to a topic, it will either stick to a certain partition number (example: partition 1, partition 2, etc.) or it will randomly choose a partition. A consumer will then subscribe to a topic and will automatically be assigned to one or more partitions by Kafka. The consumer will then start receiving messages from the assigned partitions. One very important aspect to note is that **the number of consumers within a consumer group must not exceed the number of partition for a given topic**. If you have more consumers than number of partitions, then the consumers will be hanging with no messages to consume. | ||
|
|
||
| Each messages in a topic will then have an "offset" (number), this would easily translates to "index" in an array. The offset will be used by the consumer to track where it is in the log, and what's the last message it has consumed. If the consumer is not able to keep up with the producer, it will start to lag behind. Most of the times, we want "lag" to be as low as possible, meaning we don't want to have so many unprocessed messages. The easy solution would be adding more partitions and increasing the number of consumers. |
There was a problem hiding this comment.
Do you want to mention that offsets are scoped to a partition, and that each partition in a topic will have the same offset numbers?
There was a problem hiding this comment.
Yes I should be mentioning that
|
|
||
| This happens where Kafka and the consumers get out of sync. Possible reasons are: | ||
|
|
||
| 1. Running out of disk space or memory |
There was a problem hiding this comment.
I could be wrong, but I think if kafka runs out of disk space the service crashes (which wouldn't cause offset out of range on consumers)
There was a problem hiding this comment.
Technically speaking, you're right. But after a disk out of space incident, there will potentially be a massive offset out of range error.
Perhaps this should be made clearer.
| 1. Running out of disk space or memory | ||
| 2. Having a sustained event spike that causes very long processing times, causing Kafka to drop messages as they go past the retention time | ||
| 3. Date/time out of sync issues due to a restart or suspend/resume cycle | ||
| When a producer sends a message to a topic, it will either stick to a certain partition number (example: partition 1, partition 2, etc.) or it will randomly choose a partition. A consumer will then subscribe to a topic and will automatically be assigned to one or more partitions by Kafka. The consumer will then start receiving messages from the assigned partitions. One very important aspect to note is that **the number of consumers within a consumer group must not exceed the number of partition for a given topic**. If you have more consumers than number of partitions, then the consumers will be hanging with no messages to consume. |
There was a problem hiding this comment.
Adding on this, if messages don't have partition keys, kafka will assign the message to a partition via round-robin (so technically not randomly)
Co-authored-by: Shannon Anahata <[email protected]>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Co-authored-by: Shannon Anahata <[email protected]>
Co-authored-by: Shannon Anahata <[email protected]>
Co-authored-by: Shannon Anahata <[email protected]>
…ing-kafka' into aldy505/self-hosted/troubleshooting-kafka
DESCRIBE YOUR PR
Turns out most of our self-hosted users has never touch Kafka before, so it's a good idea to introduce them regarding how Kafka works.
Also added how to increase consumers replica if they're lagging behind.
IS YOUR CHANGE URGENT?
Help us prioritize incoming PRs by letting us know when the change needs to go live.
SLA
Thanks in advance for your help!
PRE-MERGE CHECKLIST
Make sure you've checked the following before merging your changes:
LEGAL BOILERPLATE
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.
EXTRA RESOURCES